Augmented reality maintenance system with flight planner

ABSTRACT

An enhanced reality maintenance system for operating in a hazardous or inaccessible environment employs an environment modeler which is supplied with spatial parameters of a given environment and creates a computer model of the environment. An environment renderer creates a plurality of images, each corresponding to a viewing location and orientation, `viewpoint`. A remotely operated vehicle (ROV) attached to a base unit by a tether cord or radio link navigates in the environment. The ROV has a spatial imaging device, such as a video camera, and actuators which propel it through the environment. Its position and orientation are determined by a position and attitude (P&amp;A) sensing unit, and are passed to an ROV renderer which creates an image of a prestored model of the ROV having the same location and orientation as the ROV and viewed from a specified viewpoint. The viewpoints may be predetermined, provided to the system or may be interactively determined as an offset from the ROV position and orientation. Alternative embodiments include an image archive and comparison unit capable of storing images linked to information of the image acquisition, retrieving stored images with the image acquisition into and transforming one of the images to match the image acquisition information of the other image.

CROSS-REFERENCE TO RELATED APPLICATIONS

Related to U.S. patent applications "Augmented Reality MaintenanceSystem" U.S. Ser. No. 08/219,565 filed Mar. 29, 1994 and "AugmentedReality Maintenance System with Archive and Comparison Device" U.S. Ser.No. 08/219,562, filed Mar. 29, 1994 U.S. Pat. No. 5,412,569 by Nelson R.Corby, Jr., Peter M. Meenan, Claude H. Solanas, David C. Vickerman,Christopher A. Nails; and "Augmented Reality Maintenance SystemEmploying Robotics Arm" U.S. Ser. No. 08/219,561, filed Mar. 29, 1994 byClaude H. Solanas, Nelson R. Corby, Jr., Peter M. Meenan, David C.Vickerman, all filed with this application and all assigned to thepresent assignee.

CROSS-REFERENCE TO RELATED APPLICATIONS

Related to U.S. patent applications "Augmented Reality MaintenanceSystem" U.S. Ser. No. 08/219,565 filed Mar. 29, 1994 and "AugmentedReality Maintenance System with Archive and Comparison Device" U.S. Ser.No. 08/219,562, filed Mar. 29, 1994 U.S. Pat. No. 5,412,569 by Nelson R.Corby, Jr., Peter M. Meenan, Claude H. Solanas, David C. Vickerman,Christopher A. Nails; and "Augmented Reality Maintenance SystemEmploying Robotics Arm" U.S. Ser. No. 08/219,561, filed Mar. 29, 1994 byClaude H. Solanas, Nelson R. Corby, Jr., Peter M. Meenan, David C.Vickerman, all filed with this application and all assigned to thepresent assignee.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a remotely operated vehicles and morespecifically to computer enhanced environment visualization of aremotely operated vehicle.

2. Description of Related Art

Typically there is a need for inspection of machines or structures whichare in environments which are inaccessible or very hazardous for humans.Several such environments would be inside a nuclear reactor boiler, deepbeneath the sea, in a forest fire, in an oil well or in an areacontaminated with a poisonous gas. The high temperatures, radiationexposure, high pressure, or toxic effects of these environments areclearly dangerous for humans. The space requirements to examine anin-line pump within an oil well or other environments with limited spacealso preclude humans access.

Typically these machines and structures within these environments havebeen inspected or repaired by remotely operated vehicles (ROV). TheseROVs may be attached to a control unit some distance away by a tethercord or may be run by radio signals from a control unit at a safedistance away. The ROVs typically have a method of sensing theirenvironment, with a testing or imaging device, such as a video camera.ROVs also employ a means of propelling themselves around theirenvironment. In a fluid, like water, it may be a number of propellersdriven by electric motors.

The use of ROVs also typically require a method of determining theposition and orientation of the ROV (and/or its subparts) with respectto the operating environment allowing it to successfully move the ROVthrough the inspection environment. An example of a position sensingsystem employs SONAR operating through the water of the nuclear reactorboiler. Conventional systems require complicated compensation schemesand frequent recalibration to offset the errors due to variations ornoise in the environment. For example, the time of flight of a SONARsignal depends on the temperature of the water through which the SONARpulse travels. Temperature gradients within the pressure vessel must becarefully mapped and monitored to allow accurate position determination.

Typically the ROV will carry a number of inspection sensors. Typicalsensors include underwater TV cameras, ultrasound flaw detectiontransducers, thermal imagers and point probes, such as microphones.

The major problem in the use of ROVs for inspection and repair in theseremote environments is the difficulty of accurately positioning the ROVat desired locations within a complicated environment and then verifyingthat position and orientation, and passing the position and orientationto persons analyzing data from the ROV or other support personnel.

Another problem occurs as the ROV is moved from one site to anothermanually within the environment. In this situation, it is difficult toaccurately determine the ROV's position at a given instant. Since one ofthe sensors typically carried is an underwater TV camera, the operatorwill often try to use the video from the camera to determine the exactposition and orientation of the ROV, especially when the camera is notfacing in the direction the ROV is moving. Typically the operator willzoom the camera back to wide angle and may move the ROV further awayfrom a particular feature in an attempt to determine where in theenvironment he actually is. This task is made easier to the extent theposition and orientation sensing system is accurate and reliable. Often,the P&A systems are not very accurate and it may take a long time toaccurately position the ROV for inspection or repair.

ROVs are typically used in determining cracks and fractures insideenvironments, such nuclear reactor boilers. Several problems arise usingROVs and nuclear reactor boilers. One problem is that irregularitiesneed to be monitored over a period of time (on the order of years) todetermine the rate of deterioration. Presently this is accomplished bymoving an ROV to a particular position and videotaping the structure ordevice which is to be examined. At a later date the ROV is positioned atthe same site and current data (such as a video image) is compared toprevious data. Since it is very difficult to position the ROV at exactlythe same site and orientation in three dimensions and obtain a videoimage from exactly the same viewpoint as previous times, it is difficultto determine differences between images. This tends to be a verysubjective determination being made by the operator. The actual cost ofmaintenance of a nuclear power facility is not only related to the costof inspection, but is also due to the time that the plant is off-line.This typically can be many times the actual cost of maintenance. It istherefore beneficial to complete inspection and repair in a minimum timeperiod.

A related problem that affects the speed and accuracy of the inspectionhas to do with the difficulty of retrieving all pertinent past data. Ifan operator is reinspecting a given location in the reactor, he needsall past information that relates to that site. This may consist ofstill imagery, segments of past videotapes of a site, auxiliary sensordata such as ultrasound and thermal images as well as non-image datasuch as written reports and observations or perhaps audio taperecordings of sounds at the site. If this background information isscattered over many physical locations and is recorded or stored on manytypes of media, (paper, photos, handwritten notes, audio tapes, magneticvideo tapes or discs etc) it becomes very difficult to rapidly makeinspection decisions.

Another problem which arises in inspecting or examining structures withan ROV is that of planning the actual trajectory of the ROV needed tomove it from one site to the next. The environment typically has objectswhich the ROV must avoid when traveling from one point to another.Currently, an operator examines environment blueprints, and with hisknowledge of the ROV size and shape, maneuvers the ROV through theenvironment. It is very difficult to visualize the full complexity ofthe 3D environment and whether a given pathway actually will allowpassage of the real ROV. Since control of the ROV is complex anddemanding, it becomes a very difficult task for the operator to "sizeup" the suitability of a given approaching pathway while trying tophysically control the progress of the ROV.

Currently, there is a need for a system which can provide efficientremote inspection and repair in inaccessible or hazardous environments.

SUMMARY OF THE INVENTION

A remote maintenance system employs an environment modeler for receivingparameters defining objects in a hazardous or inaccessible environment.The environment modeler produces a computer generated model of theenvironment.

A remotely operated vehicle (ROV) carrying a sensor package, for imagingand performing tests on structures in the environment, and actuators,capable of maneuvering the ROV, is placed in the environment.

A position and attitude (P&A) sensing unit senses the position andorientation of the ROV relative to the environment.

A viewpoint for rendering a computed view of the environment isdetermined by the operator (either by inputting viewpoint coordinates orby selecting from a pre-established list of viewpoint coordinates) andprovided to an environment renderer which generates an imagecorresponding to the shapes defined by the environment geometry whenviewed from the selected viewpoint.

The position and orientation of the ROV is provided to the environmentrenderer along with offsets which define the position and orientation ofthe sensors with respect to the ROV. The resultant viewpoint, (formed bycombining the position and orientation of the ROV and an offsetdisplacement of the sensors) will allow the environment renderer toproduce images corresponding to views of the environment as "seen" fromthe viewpoint of the sensor package. The imagery produced by theenvironment renderer will vary in real-time according to the positionand orientation of the ROV.

Similarly, position and orientation of the ROV is provided to an ROVrenderer (along with operator indicated viewpoint). The ROV renderergenerates an image of the ROV as seen from the same viewpoint used bythe environment renderer.

A video mixer superimposes the image of the ROV on the image of theenvironment and displays the superimposed images on a monitor, therebyallowing an operator to visualize the position of the ROV relative toits environment.

Several viewpoints and superimposed images may be producedsimultaneously to provide multiple views of the ROV in the environment.

In an alternate embodiment, sensory data from the ROV sensor package isstored along with auxiliary information such as the spatial location ofthe sensors and parameters employed in acquiring the sensor data andsensor images. Any of these past images or past sensor data may later berecalled and transformed (if necessary) so as to correspond to thecurrent position and orientation of the ROV and its sensors. Digitalsignal processing techniques may then be performed to determine the rateof corrosion or rate of crack growth over time, a very importantparameter for nuclear reactor maintenance. In addition, a signalprocessor and visualization unit allows current or past imagery fromother modalities such as ultrasound scans, to be merged with current orpast video imagery. Pertinent data, such as past inspection results andoperator observations, are also extracted from the inspection databaseautomatically and displayed for operator consideration.

In another alternate embodiment, a pointing device is provided for theoperator to select a trajectory, being a time-ordered sequence oflocations to be visited by the ROV. Environment geometry, ROV geometryand the inspection itinerary, are provided to a path execution unit tocause ROV actuators to move ROV according to the desired trajectory.

OBJECTS OF THE INVENTION

It is an object of the present invention to provide a system forvisualizing the position and orientation of a remotely operated vehiclewithin a specified environment.

It is another object of the present invention to provide a remotelyoperated vehicle which automatically navigates through a hazardousenvironment crowded with objects.

BRIEF DESCRIPTION OF THE DRAWING

While the novel features of the invention are set forth withparticularity in the appended claims, the invention, both as toorganization and content, will be better understood and appreciated,along with other objects and features thereof, from the followingdetailed description taken in conjunction with the drawings, in which:

FIG. 1 is a simplified block diagram of an enhanced reality systemaccording to the present invention.

FIG. 2 is a more detailed block diagram of automated flight planner ofFIG. 1.

FIG. 3 is a more detailed block diagram of the archive and comparison(A&C) device of FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a simplified block diagram of the major components of anaugmented reality maintenance system according to the present invention.

Parameters defining an environment, such as blueprint measurements,position and shape measurements, and material types may be manuallyprovided to an environment modeler 39. These parameters may also beautomatically provided by automated spatial measurement from ultrasoundor laser distance measurement devices. Environment modeler 39 constructsa computer model from the parameters it receives which may be rapidlymanipulated in near-real-time. The model is stored in environment modelstorage device 37.

Parameters defining the geometry of the ROV are provided to an ROVmodeler 49. ROV modeler 49 constructs a computer model from theparameters it receives. The model of the ROV is stored in ROV modelstorage device 47.

An environment renderer 35 has the capability of accessing the model inenvironment model storage device 37, and displaying it as viewed fromany location and orientation, known as a `viewpoint`. It also has thecapability of creating several displays of the model viewed from severaldifferent viewpoints simultaneously.

A remotely operated vehicle (ROV) 10, attached by tether cord, or aradio link to a base unit, is intended to be placed in a hazardous orinaccessible environment and employs a plurality of actuators 13 whichpropel the ROV through the medium of the environment. In a nuclearreactor boiler, filled with water, the actuators are electric motorsattached to propellers. These may move the ROV in a number of directionsand rotate it in any orientation. In a terrestrial environment, theactuators are motor driven wheels, tracks or belts.

A sensor package 11 is attached to the ROV. This acquires informationabout desired structures or devices. Spatial imaging device, in its mostcommon form is a video camera, acquires information such crack andcorrosion in a wall of a nuclear reactor boiler, for example. Sensorpackage 11 may also be an ultrasound device capable of detectingirregularities in a structure or device, or any such modality used instructure examination. Sensor package 11 could also be a pointmeasurement probe such as a microphone or accelerometer to measurevibrations. Also, more that one sensor could be operating at any time.

A position and attitude sensing (P&A) unit 21 determines an approximatelocation and orientation of the ROV. This may be, for example, SONARsound sources, dispersed at known locations around the environment whichsuccessively emit a signal. Multiple sensors on the ROV at knownlocations sense the signal from the SONAR sources. A receiver attachedto the sensors determines the location (x,y,z) and orientation (α, β, γ)of the ROV based upon differences in the time of reception of the signalby each sensor.

The position (x,y,z) and orientation (α, β, γ) determined by P&A sensingunit 21 is provided to an ROV renderer 33. ROV renderer 33 creates animage of a prestored model of the ROV from model storage device 47, atlocation (x,y,z) and orientation (α, β, γ) as viewed from an viewpointprovided to it. ROV renderer 33 has the capability of creating severalimages of the ROV model viewed from several different viewpointssimultaneously. The position and orientation of the ROV, produced by P&Asensing unit 21, are supplied to offset computation device 31a. Thisdevice calculates an new viewpoint which is offset from the origin ofthe ROV model. This is used to define viewpoints corresponding to thesensors carried by the ROV.

The environment renderer 35 produces a number of images of theenvironment corresponding to viewpoints supplied to it. In FIG. 1, twoviewpoints are supplied--one from the viewing viewpoint unit 31b andanother from the offset computation device 31a. The environment renderer35 produces two image signals, one showing the environment viewed from aset viewpoint and a second showing the environment as seen from theviewpoint of the selected sensor travelling with the ROV.

The viewpoint provided to ROV renderer 33 and environment renderer 35may be predetermined values which have been typed into, or are residentin, viewpoint unit 31b.

The images pertaining to the same viewpoint from ROV renderer 35 andenvironment renderer 45 are provided to a video mixer 41. In theembodiment shown in FIG. 1, two viewpoints are employed, with two imagesproduced by environment renderer 35 and one by the ROV renderer 33. TheROV image and the environment image for a single viewpoint are providedto a video mixer, to superimpose the ROV image on the environment imageresulting in a superimposed image showing the relative position of theROV model in the simulated environment, which corresponds to the actualposition of the ROV relative to the actual environment. Video mixer 41receives images from environment renderer 35, the ROV renderer 33 andthe sensor package 11 carried by the ROV 10. Video mixer 41 produces animage for monitor 43 for viewing by operator 2. The image produced onmonitor 43 may consist of a number of sub-images comprising the mainimage. One such sub-image is the result of merging the image of theenvironment produced by the environment renderer 35 with the image ofthe ROV produced by the ROV renderer 33. The viewpoint for bothrenderers is the same and is supplied by viewpoint unit 31b. The secondsub-image may be a view of the environment as seen from the point ofview of the sensor package 11 travelling with the ROV 10. The thirdsub-image appearing on monitor 43 is an image produced by the sensorpackage 11 carried by ROV 10.

in alternative embodiments, more or fewer viewpoints, images may be usedto create more or fewer superimposed images. Also, in anotheralternative embodiment, an operator may select or change the viewpointsand the offsets during operation.

In another embodiment, the present invention further employs anautomated flight planner 60 which is coupled to environment modelstorage device 37, ROV model storage device 47 and actuators 13. FIG. 2is a more detailed block diagram of automated flight planner 60 ofFIG. 1. In FIG. 2, automated flight planner 60 is composed of anpointing device 65, a flight path storage device 61 for storing intendedpaths, or trajectories, of ROV 10, a path execution unit 67 and atrajectory computer 63. Pointing device 65 is used by operator 2 tochoose destination locations in the environment which the ROV is tovisit, which are provided to trajectory computer 63. Trajectory computerwill then read information from ROV model storage unit 47 andenvironment model storage unit 37 and calculate a trajectory to thedestination point which would not cause a collision with any objects inthe environment. The computed trajectory may then be displayed to theoperator on monitor 43 via video mixer 41 of FIG. 1. Operator 2 may alsodefine destination points by typing symbolic names, or numericallocations into pointing device 65. Operator 2 may determine that thetrajectory is acceptable and cause path execution unit 67 to execute thetrajectory. Path execution unit 67 drives actuator 13 (FIG. 1) to movethe ROV according to the calculated trajectory. Thus, the currentcalculated trajectory (or one selected from flight store device 61) canbe executed by the actual ROV within the actual environment upon commandby operator 2. Optionally, a flight path monitoring unit 69 reads theselected trajectory from path storage device 61, and receives thecurrent position and orientation of the ROV, and indicates on monitor43, via video mixer 41, the selected trajectory and the current pathfollowed by the ROV.

Operator 2 may indicate not only destination points, but intermediatepoints of the trajectory. The trajectory computer determines atrajectory or appropriate path nearest the points indicated by operator2 which will not cause a collision.

Since rapid, accurate determination of irregularities is very importantin many cases, such as in off-line maintenance of a nuclear power plant,and costs are related to the length of time the plant is off-line, it isimportant to collect, retrieve and compare image data rapidly. Inconventional systems, video images are acquired of suspect sites of astructure. At a later date, the ROV is directed manually to one of thesuspect sites. A current video image is acquired, and the images areviewed side-by-side, usually in real-time, to determine the degree ofdeterioration. The comparisons are only valid when the archived imageand the current image have the similar imaging parameters. Imagingparameters vary from one modality to another. For example, video camerasimaging parameters include the viewpoint, field-of-view, iris opening,zoom setting etc. By varying these parameters, the image becomesdifferent.

The information from past inspections at a given site may take manyforms such as photographs, video frames, video sequences on videotape,computer generated images which visualize data such as 2D ultrasonicinspection data, thermal imagery as well as inspectors reports and notesand non-image data e.g. audiotapes.

In another embodiment of the present invention, the invention comprisingall of the previously described elements of FIG. 1, with automatedflight planner 60 being optional, further comprises an archive andcomparison (A&C) device 50. FIG. 3 is a more detailed block diagram ofthe A&C device of FIG. 1. In FIG. 3, A&C device 50 utilizes a sensordata storage device 51, capable of storing spatial imagery withlocation, orientation and acquisition parameters linked to each image.These parameters define the identity of the site imaged, when it wasimaged, the viewpoint, the modality of the imager (visual, thermal,ultrasonic etc.) and description of values relating to the image (cracklength, corrosion area etc.). In addition, storage device 51 providesstorage for textual information such as inspectors reports and storageof non-image signal data such as recordings of microphones oraccelerometers carried by the ROV. The textual data, and non-imagesignal data, also are linked to specific inspection sites and timestamped for identification at a later retrieval time. Much of theinformation provided to storage device 51 originates in sensor package11 carried by the ROV 10. Textual information may be provided by a textinput device 57.

A&C device 50 also includes a retrieval control computer 53, coupled tothe data storage device 51, the P&A unit 21, the signal processing unitand visualization (SP&V) unit 55. Retrieval control computer 53, uponcommand by operator 2, retrieves all past data from storage device 51which is pertinent to the site currently being examined and visited bythe ROV. SP&V unit 55 receives sensor data from sensor package 11 andpast inspection data from storage device 51 under control of theretrieval control computer 53. SP&V unit 55 transforms images archivedin sensor data storage device 51, according to the position, orientationand imaging parameters, to match those of images currently beingacquired by sensor package 11. The signals may then be placed on thesame basis for comparison. SP&V unit 55 may either display the twoimages to operator 2 via video mixer 41 on monitor 43 in a side-by-sideformat, superimpose them, display image differences or employ anyappropriate image processing methods thus highlighting regions forspecial attention by operator 2. The differences may be highlighted bycolor coding, graphical display etc. SP&V unit 55 may also display itsresults of comparisons and image processing in any appropriate form forconsideration by operator 2.

SP&V unit 55 may also operate on non-image signals, such as sound data,to cause two signals to have the same acquisition parameters, andperform comparisons and signal processing on the transformed signals.

Retrieval control computer 53 may select two archived images to compareagainst each other instead of one archived and one current image. SP&Vunit 55 will transform, one image, the other image, or both, to have thesame viewpoint and imaging parameters allowing them to be directlycompared. A number of archived images for the same site acquired atdifferent times, may be transformed by the SP&V unit 55, to compose a`time-lapse` movie when they are played back a time-ordered sequence.

Many elements of the block diagram of FIG. 1 may be physically locatedin the ROV or in the base unit, making little difference where they arelocated, except that monitor 43 and pointing device 65 must beaccessible to operator 2; and actuators 13 and sensor package 11 must beon the ROV.

While several presently preferred embodiments of the present novelinvention have been described in detail herein, many modifications andvariations will now become apparent to those skilled in the art. It isour intent therefore, to be limited only by the scope of the appendingclaims and not be the specific details and instrumentalities presentedby way of explanation herein.

What we claim is:
 1. A remote maintenance system for inspection andrepair of structures in an environment comprising:a) a remotely operatedvehicle (ROV) having1. a sensor package, capable of inspecting physicalcharacteristics of said structures in environment, and
 2. actuatorscapable of maneuvering the ROV; b) a position and attitude (P&A) sensingunit for providing a position and orientation of the ROV; c)environmental model storage device capable of retaining computer graphicinformation regarding the environment of the ROV; d) environment modelerfor receiving parameters defining objects in an environment of the ROVand producing a computer generated model of the environment and storingthe model in the environment model storage device; e) environmentrenderer coupled to the environment model storage device for generatingan image of the environment from the environment model as viewed from atleast one viewpoint; f) ROV renderer coupled to the P&A sensing unitadapted for generating an image of the ROV at a position and orientationcorresponding to that of the ROV, generated from a prestored model ofthe ROV as viewed from at least one viewpoint; g) monitor for displayinga video signal; h) video mixer coupled to the environment renderer andthe ROV renderer for displaying the images on the monitor, therebyallowing an operator to visualize the position of the ROV relative toits environment; and i) an automated flight planner comprised of:i. apointing device allowing the operator to select points on a path whichthe operator desires the ROV to follow; ii. a flight path storage unitcapable of storing information defining a path through space; and iii. atrajectory computer coupled to the environment model storage device, thepointing device, the flight path storage unit, functionally coupled tothe ROV renderer and the actuators, for approximating a path closest tothe points selected by the operator which does not cause the ROV tocollide with objects in the environment, being a selected trajectory,storing this trajectory in the flight path storage unit, and causing theROV to follow the selected trajectory.
 2. The remote maintenance systemof claim 1 further comprising viewpoint unit coupled to the ROV rendererand the environment renderer which provides the viewpoint employed inimage rendering.
 3. The remote maintenance system of claim 1 furthercomprising offset computation device coupled to the P&A sensing unitwhich provides the viewpoint, being an offset to location andorientation of the ROV received from the P&A sensing unit, employed bythe environment renderer and the ROV renderer in image rendering.
 4. Theremote maintenance system of claim 1 further comprising a path executionunit coupled to the P&A unit and the flight path storage unit andactuators for reading the selected trajectory from the flight pathstorage unit and causing the actuators to move the ROV along that path.5. The remote maintenance system of claim 1 further comprising a flightpath monitoring unit coupled to the flight path storage unit, the P&Aunit and the video mixer, for reading the selected trajectory from theflight path storage unit, and monitoring the actual location andposition of the ROV over time, indicating the ROV's actual path, anddisplaying the selected trajectory and the actual path of the ROV on themonitor.